An Information Theoretic Tradeoff between Complexity and Accuracy
نویسندگان
چکیده
A fundamental question in learning theory is the quantification of the basic tradeoff between the complexity of a model and its predictive accuracy. One valid way of quantifying this tradeoff, known as the “Information Bottleneck”, is to measure both the complexity of the model and its prediction accuracy by using Shannon’s mutual information. In this paper we show that the Information Bottleneck framework answers a well defined and known coding problem and at same time it provides a general relationship between complexity and prediction accuarcy, measured by mutual information. We study the nature of this complexity-accuracy tradeoff and discuss some of its theoretical properties. Furthermore, we present relations to classical information theoretic problems, such as rate-distortion theory, cost-capacity tradeoff and source coding with side information.
منابع مشابه
Performance versus Complexity Per Iteration for Low-Density Parity-Check Codes: An Information-Theoretic Approach
The paper is focused on the tradeoff between performance and decoding complexity per iteration for LDPC codes in terms of their gap (in rate) to capacity. The study of this tradeoff is done via information-theoretic bounds which also enable to get an indication on the sub-optimality of message-passing iterative decoding algorithms (as compared to optimal ML decoding). The bounds are generalized...
متن کاملAn Investigation into the Effects of Joint Planning on Complexity, Accuracy, and Fluency across Task Complexity
The current study aimed to examine the effects of strategic planning, online planning, strategic planning and online planning combined (joint planning), and no planning on the complexity, accuracy, and fluency of oral productions in two simple and complex narrative tasks. Eighty advanced EFL learners performed one simple narrative task and a complex narrative task with 20 minutes in between. Th...
متن کاملAn Information-Theoretic Discussion of Convolutional Bottleneck Features for Robust Speech Recognition
Convolutional Neural Networks (CNNs) have been shown their performance in speech recognition systems for extracting features, and also acoustic modeling. In addition, CNNs have been used for robust speech recognition and competitive results have been reported. Convolutive Bottleneck Network (CBN) is a kind of CNNs which has a bottleneck layer among its fully connected layers. The bottleneck fea...
متن کاملThe effect of increase in task cognitive complexity on Iranian EFL learners’ accuracy and linguistic complexity: A test of Robinson’s Cognition Hypothesis
Designing a task with a reasonable level of cognitive complexity has always been important for syllabus designers, teachers, as well as researchers. This is because task manipulation may lead to different results in oral production. The present study was an attempt to explore the effect of this manipulation - based on Robinson’s resource-directing model (reasoning demands, number of el...
متن کاملInformation Theoretic Combination of Classifiers With Application to Multiple SVMs
Combining several classifiers has proved to be an effective machine learning technique. Two concepts clearly influence the efficiency of an ensemble: the diversity between classifiers and the individual accuracies of the classifiers. In this paper we propose an information theoretic framework to establish a link between these quantities and, as they appear to be contradictory, we propose an inf...
متن کامل